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ABSTRACT 

The Kepler planet sample can only be used to reconstruct the underlying planet occurrence rate 
if the detection efficiency of the Kepler pipeline is known; here we present the results of a second 
experiment aimed at characterising this detection efficiency. We inject simulated transiting planet 
signals into the pixel data of ~10,000 targets, spanning one year of observations, and process the 
pixels as normal. We compare the set of detections made by the pipeline with the expectation from 
the set of simulated planets, and construct a sensitivity curve of signal recovery as a function of 
the signal-to-noise of the simulated transit signal train. The sensitivity curve does not meet the 
hypothetical maximum detection efficiency, however it is not as pessimistic as some of the published 
estimates of the detection efficiency. For the FGK stars in our sample, the sensitivity curve is well 
fit by a gamma function with the coefficients a = 4.35 and b = 1.05. We also find that the pipeline 
algorithms recover the depths and periods of the injected signals with very high fidelity, especially for 
periods longer than 10 days. We perform a simplified occurrence rate calculation using the measured 
detection efficiency compared to previous assumptions of the detection efficiency found in the literature 
to demonstrate the systematic error introduced into the resulting occurrence rates. The discrepancies 
in the calculated occurrence rates may go some way towards reconciling some of the inconsistencies 
found in the literature. 

Subject headings: techniques: photometric — - methods: data analysis — missions: Kepler 


1. INTRODUCTION 

The Kepler Mission is a NASA Discovery Program 
mission designed to characterise the population of plan¬ 
etary systems using high precision photometric observa¬ 
tions. The primary goal of the Kepler Mission is to mea¬ 
sure 7/0, the frequency of Earth-size planets in the habit¬ 
able zone of Sun-like stars. The spacecraft was launched 
in 2009, and for four years monitored the brightness of 
~192,000 stars nearly continuously, looking for the peri¬ 
odic dimmings indicative of transiting planets. The Ke¬ 
pler project has produced several planet candidate cat¬ 
alogues (Borucki et al. 2011a,b; Batalha et al. 2012; 
Burke et al. 2014; Rowe et al 2015; Mullally et al. 2015) 
from these data. 

In order to determine the real, underlying popula¬ 
tion of planets from a sample of planet candidates, 
an essential step is to quantify the false negative 
rate: the fraction of the real planets that should 
have been detected that are not included in the sam¬ 
ple (also called the survey completeness or survey de¬ 
tection efficiency). Initial analyses of the published 
Kepler planet candidate catalogues, e.g. Borucki 
et al. (2011b; r eferred to as B ll fo r the remain¬ 
der of t his paper!. iCatanzarite fc Shaol (I2011IL lYoudinl 
(l2011li. iHoward et al.l (I2012U . iDong fe Zhul (|2013lf and 
iFressin et al.l ( 201311 used various estimates of detection 
efficiency to constrain the occurrence rate of planets, 
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but as yet there is no definitive empirical measure of 
this value for the Kepler pipeline. Petigura, Howard & 
Marcy (2013; referred to as PHM13 for the remainder 
of this paper) used a custom-built pipeline to produce 
their own planet candidate sample from the Kepler data, 
which allowed them to directly quantify their detection 
efficiency and remove the uncertainties caused by esti¬ 
mating this quantity. We have initiated a large, robust 
study to empirically measure the false negative rate in 
the Kepler planet c andid ate sample, first introduced in 
I Christiansen et all (I2013T ) . hereafter referred to as Pa¬ 
per I. In that study, we investigated the ability of the 
pipeline to preserve individual transit events, finding an 
extremely high fidelity of 99.7% recovery of the expected 
signal strength across most of the investigated parameter 
space. This is extremely valuable for people performing 
their own transit searches using the Kepler light curves, 
as they can be assured that there has been little to no 
corruption of the signals at that point (barring the tran¬ 
sits of very short period planets with periods below three 
days, or transits falling within two days of a gap in the 
Kepler data, as demonstrated in Paper I). 

The next step is to investigate the ability of the 
pipeline to recover periodic transit signals, as compared 
to individual transit signals. Here we present the results 
of our measurement of the detection efficiency of the Ke¬ 
pler pipeline across one year of long-cadence (30-minute 
integration) observations, comprising Q9-Q12 (for a de¬ 
tailed review of the Kepler Mission design, performance 
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and data products, see Koch et al. 2010, Borucki et al. 
2010 and Jenkins et al. 2010a). To measure the detection 
efficiency, we injected the signatures of simulated tran¬ 
siting planets into the calibrated pixels of ~26,000 target 
stars across the focal plane, processed the pixels through 
the data reduction and planet search pipeline as normal, 
and examined the resulting detections. In Section [2j we 
describe in detail the configuration of the tested pipeline 
and the generation and injection of the simulated planet 
signals. In Section[3]we examine the characteristics of the 
detections and generate the pipeline’s sensitivity curve. 
In Section[4]we explore the impact of the sensitivity curve 
on the derived underlying planet population, in order 
to understand the systematic biases in the derived oc¬ 
currence rates caused by assumptions about the pipeline 
sensitivity. Finally in Section[5]we summarise the results 
and outline the further work required. 

2. EXPERIMENT DESIGN 

The data reduction pipeline has been d escribed i n de¬ 
tail in a series of papers; for an overview see I Jenkins et all 
•: 2010i> : and Figure 1 therein. The performance of the 
‘front end’ of the pipeline, comprising the modules Cali¬ 
bration (CAL: calibration of raw pixels; Quintana et al. 
2010), Photometric Analysis (PA: construction of the ini¬ 
tial flux time series from the optimal aperture for each 
target; Twicken et al. 2010), and Pre-Search Data Con¬ 
ditioning (PDC: removal of common systematic signals 
from the flux time series; Smith et al. 2012, Sturnpe 
et al. 2012, Sturnpe et al. 2014), in preserving tran¬ 
sit signals was examined in Paper I. The performance 
of the full pipeline, including the ‘back end’, compris¬ 
ing the modules Transiting Planet Search (TPS: search¬ 
ing the light curves for periodic transit signals; Jenkins 
et al. 2010b, Seader et al. 2013) and Data Validation 
(DV: examination and validation of the resulting candi¬ 
date signals against a suite of diagnostic tests; Wu et al. 
2010), is examined here. In particular, the data prod¬ 
ucts and software versions match those used to produce 
the Q1-Q16 Threshold Crossing Eve nt (TCE) catalogue 
presented in lTenenbaum et~j (120131 ) and the associated 
Q1-Q16 Kepler Object of Interest (KOI) catalogue (Mul- 
lally et al. 2015). Explicitly, the CAL and PA products 
were from Data Releases 12 (8.0), 13 (8.0), 15 (8.0) and 
17 (8.1) for Quarters 9, 10, 11 and 12 respectively, where 
the SOC pipeline version is given in brackets after the re¬ 
lease number; the PDC products were from Data Release 
21 (8.3); and the TPS and DV products were produced 
using version 9.1 of the SOC pipeline. Using the more 
updated PDC and TPS/DV products in both the gener¬ 
ation of the Ql- Q16 catalogue and this transit injection 
experiment allowed us to take advantage of the inter¬ 
vening updates (including Bayesian analysis to remove 
common systematic signals across target light curves in 
PDC and vetoing of non-astrophysical signals and iter¬ 
ative searching in TPS), without waiting for the long 
term re-processing of the CAL and PA products. Subse¬ 
quently, updates to CAL and PA which have improved 
the detection efficiency in the meantime are not tested 
here; they will however be included in the Q1-Q17 cata¬ 
logue and corresponding transit injection experiment. 

Some of the potential areas for signal loss in the 
pipeline are described in Paper I. One of these is sig¬ 
nal distortion, by such processes as aperture errors and 


losses, and Sudden Pixel Sensitivity Drop-outs (SPSDs) 
occurring during transits. Additional causes of signal dis¬ 
tortion include pipeline processes, such as the systematic 
error removal in PDC or the harmonic removal in TPS. 
Another source of signal loss is caused by signal masking: 
the pipeline now iteratively searches each flux time series 
down to the 7.1<r signal detection threshold, removing the 
observations that contributed to the detected signals be¬ 
fore searching again. This effectively reduces the number 
of cadences when transits can be observed, and as a con¬ 
sequence, the detectability of additional signals in that 
time series. 

In order to investigate the impacts of signal distortion 
and signal masking, we performed the following experi¬ 
ment. We selected 15 sky groups across the Kepler fo¬ 
cal plane—a sky group refers to the set of target stars 
that fall together on the same CCD channel. As the Ke¬ 
pler spacecraft rotates around its boresight every three 
months, to keep the solar panels pointed at the sun, 
the set of stars rotates together onto a different CCD 
channel, and so on until after four rotations (one year 
of observations), the set of stars returns to the original 
CCD channel. The sky groups were selected to sample 
a range of the focal plane architecture and CCD channel 
characteristics in Q9 (in Q10-Q12, these sky groups will 
fall on other CCDs, typically with ‘average’ behaviour); 
they are listed in Table Q] For computational reasons we 
limit this first multi-quarter transit injection experiment 
to one year of data comprising Q9-Q12. This is the first 
full year of spacecraft operations without a long data gap 
due to a spacecraft anomaly, and was chosen to isolate 
the impact of the data reduction pipeline on the detec¬ 
tion efficiency; the impact of the window function due to 
long data gaps in the full data set will be investigated 
more thoroughly in the full baseline (Q1-Q17) transit 
injection experiment. 

Across these sky groups, we inject simulated transits 
into every target star. We genera te our injected transits 
using the iMandel fe Agoll (|2002T ) model. For each tar¬ 
get star, the parameters of an initial estimated transit 
model are constructed from four observable parameters: 
(1) The signal-to-noise ratio (SNR) of a single transit 
is randomly drawn from a uniform distribution between 
2cr and 20<r; (2) The transit duration is randomly drawn 
from a uniform distribution between 1 and 16 hours (in 
the pipeline, we search for transit pulses with durations 
from 1.5-15 hours); (3) The impact parameter, b, is ran¬ 
domly drawn from a uniform distribution between 0-1; 
and (4) The phase of the first injected transit is ran¬ 
domly drawn from a uniform distribution between 0-1. 
The epoch of the first transit is also required to fall in 
Quarter 9, to ensure at least one injected transit per 
star. We then, using the calculations described in Ap¬ 
pendix A of Paper I, generate a physical transit model 
which approximately reproduces those initial observed 
parameters, and from which the actual observed param¬ 
eters are measured. We assume circular orbits and no 
limb darkening when generating the transit signal. We 
have included as electronic data both the physical and 
the measured observable parameters of the final injected 
planet models. Table [5] shows an excerpt from the table 
to illustrate the contents. 

We inject the generated transit model into the cali¬ 
brated pixels for the target in question. These modified 
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Table 1 

Sky groups and corresponding Quarter 9 CCD channels used 
in the transit injection test, and a qualitative description of 
any noted features of the channel. In the other quarters used 
in this test (Q10-Q12), the sky groups will fall on other 
CCDs, typically with ‘average’ behaviour. 


Sky group 

Channel 

Description 

32 

4 

Edge of field/worst focus 

70 

10 

Variable black/bias correction" 

71 

11 

Edge of field/worst focus 

9 

13 

FGS crosstalk^ 

25 

17 

Nominal/best focus 

66 

26 

Rolling band artefacts/Moire pattern" 

84 

32 

Edge of field/worst focus 

62 

46 

Variable black/bias correction 

78 

50 

FGS crosstalk 

4 

56 

Edge of field/worst focus 

18 

58 

Rolling band artefacts/Moire pattern 

19 

59 

Nominal/best focus 

38 

62 

Rolling band artefacts/Moire pattern 

74 

70 

FGS crosstalk 

53 

81 

Start-of-line ringing^ 


a The measured bias in these channels has a much larger scatter 
from cadence to cadence than typical channels. 
h A fixed pattern of varying-magnitude electronic crosstalk in 
the science pixels from clock crosstalk caused by the Fine 
Guidance Sensor (Kepler Instrument Handbook, 2009) 
c A varying pattern (spatially and temporally) in the science 
pixels caused by a temperature-dependent resonance in the 
Local Detector Electronics circuit (Kepler Instrument 
Handbook, 2009) 

d A fixed pattern at the beginning of rows on some channels due 
to a voltage transient when parallel clocking out rows (Kepler 
Instrument Handbook, 2009) 

pixels are then processed through the pipeline as nor¬ 
mal. As in Paper I, the only departure from standard 
operations is that the motion polynomials (used for cal¬ 
culating the location of the target) and the cotrending 
basis vectors (used in the correction of systematic er¬ 
rors) are generated from a ‘clean’ pipeline run. This is to 
avoid corruption from the presence of the injected tran¬ 
sits, since the motion polynomials and cotrending basis 
vectors are generated from the data themselves, and will 
be distorted by the presence of transit signals in every 
single target. 

In summary, the final order of processing is that we ran 
the original calibrated pixels (the output of CAL) of Q9- 
Q12 through PA, PDC, and TPS, without any modifica¬ 
tion, to generate the motion polynomials, the cotrending 
basis vectors, and the root-mean-square Combined Dif¬ 
ferential Photometric Precision (CDPP) for each target. 
We then injected the simulated transit signals into the 
calibrated pixels, one planet for every target across the 
15 sky groups, and re-ran the modified pixels through 
PA, PDC, TPS and DV, utilizing the previously gener¬ 
ated information as described. 

3. RESULTS 

3.1. Detection efficiency 

The top panel of Figure [T| shows the distribution of in¬ 
jected planet parameters for all targets across the planet 
radius range 0-111?® and the planet orbital period range 
0.5-200 days; there are 10,341 planet injections in this 
parameter space, of which 9,123 are successfully recov¬ 
ered by the Kepler pipeline (shown in red), and 1,218 
are not (shown in blue). The histogram below shows the 
fraction of the injected signals that were successfully re¬ 


covered as a function of period. Note that some of the 
injected signals are not expected to be successfully recov¬ 
ered given their expected SNR or orbital period, but are 
included as we wish to probe the sensitivity of the tran¬ 
sition region between detection and non-detection. A 
‘successful’ recovery was defined as a signal being identi¬ 
fied with an epoch within 0.5 days of the injected planet 
epoch, and a period within 3% of the injected planet or¬ 
bital period. Given the baseline of the observations (372 
days) and the requirement of three transits for detection 
in the pipeline, the maximum detectable period is ~185 
days. The relatively small number of planets injected 
with radii <2i?® is due to the limiting of the SNR of 
individual transits to be >2er. Figure [2] shows the signal 
strength of the injected planets over the orbital period 
(upper panel) and planet radius (lower panel) ranges of 
the injections, measured as the expected Multiple Event 
Statistic (MES) of the signal. The MES gives the signifi¬ 
cance of the correlation between the data and a putative 
box-shaped transit signal of a given orbital period, tran¬ 
sit du ration and phase in u nits of the uncertainty in the 
data (1 Jenkins et al.|[2010bD . In the upper panel, a linear 
fit to the data is overlaid, with a slope of -0.49, demon¬ 
strating the expected dependence of the signal strength 
on \j~N , where N is the number of transits and is in¬ 
versely proportional to the orbital period. 

There are two dominant effects visible in Figure [Tj the 
first is the drop-off in detectability at very short (predom¬ 
inately <3-day) periods; the second effect is the drop-off 
in detectability at smaller radii and longer periods, which 
is the signal-to-noise detection threshold that we want to 
examine further. 

The cause of the drop-off in detectability at very short 
periods, in all examined cases, was the behaviour of the 
harmonic fitter in TPS, as described in Paper I. Before 
the flux time series is whitened and searched for periodic 
signals, a sinusoidal harmonic filter is applied to remove 
periodic stellar activity, allowi ng the pipeline to searc h 
variable stars for transit signals (jTenenbaum et al.l2012D . 
Given the artificial separation of the injected transits in 
Paper I, we were limited in our ability to characterise 
the period parameter space where the harmonic fitter 
removes signal. With this extended analysis, we find 
that the impact of the harmonic fitter results in the non¬ 
detection of ~60% of the signals we would expect to de¬ 
tect between 0.5-0.6 days, and ~40% between 0.6-1.0 
days, and drops from there to 10% at 1.5 days and 1% 
at 2.0 days. 

We now turn our attention towards the drop-off in 
detectability with decreasing signal-to-noise, in order 
to measure the true detection threshold of the Kepler 
pipeline. In the bulk of our analysis, we limit the stel¬ 
lar sample to FGK stars (4000K < T e ff < 7000K, log 
g > 4.0). This is the sample for which the project is 
currently calculating the occurrence rate of planets; vari¬ 
ability in more active or more evolved stars can impact 
detectability, as shown below. There are 8,579 transit in¬ 
jections around the FGK sample across the planet radius 
range 0-ll.R® and the planet orbital period range 0.5- 
200 days. The bottom two panels of Figured] shows the 
distribution of injected and successfully recovered planet 
parameters for the FGK stars; 7,696 of the 8,579 injec¬ 
tions were recovered. The histogram shows the fraction 
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Figure 2. The signal strength, measured in expected MES, of the 
injected transiting planets. In both panels, the blue points show 
the signals that were not successfully recovered, and the red points 
show the recovered signals. Upper: The correlation with orbital 
period, following the expected y/~N dependence, where N is the 
number of transits; Lower : The correlation with planet radius. 


MES and the detection threshold, i.e., there is a 50% 
chance of detecting a transit signal with a MES at the 
threshold (MES=7.1er), an 84% chance of detection a 
transit sequence 1 sigma above the threshold (MES = 
8.1 <t), etc. This is shown in Figure [ 3 ] as the solid red 
curve. However, the fraction of injected transit signals 
recovered at 7.1er is only ~25%, and falls well below the 
theoretical curve for MES 6-17o\ The measured signal 
recoverability of the pipeline is well-characterised by a T 
cumulative distribution function, which has the form: 

X 

p = F(x\a,b) = ^ ) Jt"-'e- t />‘dt (1) 

0 


Figure 1. The distribution of parameters of the injected and re¬ 
covered transit signals. Upper two panels: For all targets; Lower 
two panels: For the FGK targets. In both scatter plots, the blue 
points show the signals that were not successfully recovered, and 
the red points show the recovered signals. The histograms show 
the fraction of recovered signals as a function of period. 

of the injected signals that were successfully recovered 
as a function of period; note here that the attenuation 
in detectability at short periods is mitigated somewhat. 
The attenuation in the full sample is somewhat driven by 
the more variable photometry of the giant stars on these 
short period timescales. 

The upper panel of Figure [3] shows the fraction of in¬ 
jected transit signals recovered in the FGK sample as a 
function of the expected MES. The detection threshold 
in the pipeline is a MES of 7.1er, shown in Figure [ 3 ] as 
the vertical black dotted line. The theoretical perfor¬ 
mance of the pipeline as a perfect detector dealing with 
broadband coloured Gaussian noise as a function of the 
strength of a signal’s MES (in cr), is defined by the nor¬ 
mal error function in terms of the difference between the 
MES and the detection threshold. That is, the detection 
probability of a signal is defined by the cumulative distri¬ 
bution function for a unit variance, zero-mean Gaussian 
process evaluated at the difference between the signal’s 


The best-fit coefficients to the sensitivity curve are 
a = 4.35, b = 1.05, shown as the green dashed curve. 
This corresponds to a 25.8% recovery rate of 7.1er sig¬ 
nals (compared to the 50% theoretical recovery rate), 
and this is the sensitivity curve we use in the remain¬ 
der of our analysis. In comparison, we show the de¬ 
tection efficiency of the remaining targets in the stellar 
sample (i.e., those with T e g < 4000K or > 7000K, or 
log g < 4.0) in the lower panel of Figure [ 3 ] There are 
1,762 injections, of which 1,542 were successfully recov¬ 
ered (the substantially lower number of trials results in 
the reduced smoothness of the histogram, compared to 
the FGK sample). As suspected, these stars have a lower 
detection efficiency for the same transit signal strength, 
which is likely due to increased masking of real signals 
by the presence of correlated astrophysical signals in the 
light curves. The best-fit coefficients to a fit of the T cu¬ 
mulative distribution function, shown again as the green 
dashed curve, are a = 4.77, b = 1.24, corresponding to 
a 12.0% recovery rate at 7.1er. Taking the whole stellar 
sample (10,341 total injections) results in best-fit coef¬ 
ficients of a = 4.21, b = 1.13, a 23.6% recovery rate at 
7.1u. Table [ 3 ] summarises the results. 

Briefly examining the population of signals that are not 
recovered by the pipeline is useful for identifying those 
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Non-FGK dwarfs 



Expected MES (<r) 

Figure 3. The fraction of injected signals successfully recovered 
by the pipeline, for the FGK dwarfs (4000K < T e ff < 7000K, log 
g > 4.0; 8,579 injections in total) in the upper panel and the re¬ 
maining targets (M-dwarfs and evolved stars; 2,092 injections in 
total) in the lower panel. In both panels, the black dotted line is 
the 7.1cr pipeline threshold; the red curve is the optimum detection 
efficiency of the pipeline for white noise; the green curve is the F 
function fit to the data ; and the magenta l ine is the linear detec¬ 
tion efficiency used by IFressin et all (1201311 . The increased noise 
in the lower panel is due to the significantly lower number of trial 
injections. 

processes in the pipeline that are implicated. Of the 
1,218 injections that are not recovered, 397 have an ex¬ 
pected MES of greater than 15. Only 95 of these have 
periods longer than 3 days (below which we can attribute 
the non-detections to the harmonic fitter described ear¬ 
lier) and 17 of these are recovered at an integer multiple 
of the injected orbital period. Of the remaining 78 sig¬ 
nals, 46 (~ 60%) show very strong stellar variability, stel¬ 
lar rotation or eclipsing binary signals; these light curves 
may not be treated well by the harmonic fitter, and/or 
may generate many spurious detections that cause the in¬ 
jected signal to be missed. Of the remaining 32 signals, 
26 have fewer than six transits. Here, a significant fac¬ 
tor contributing to the non-recovery of transit signatures 
is the vetoes used by TPS to discriminate between gen¬ 
uine transit signals and those caused by systematic noise 
(Seader et al. 2013, Tenenbaum et al. 2014, Seader et al. 
2015). In this version of TPS, there is a documented issue 


that the y 2 veto is overly aggressive toward signals with 
a low number of transits, due to the test not taking into 
account the mismatch in shape between the transit signal 
in the data and the modelled transit template (Seader et 
al. 2015f]. Additionally, for transiting planets where the 
individual events have low SNR, the shape of the transit 
can be distorted by the noise present in the data to the 
extent that the transit signal can fail the vetoes. Careful 
tuning of the vetoes has been performed during pipeline 
testing to preserve as many of the ‘real’ transit signals 
as possible while eliminating many tens of thousands of 
spurious signals, but we can see the impact of the ve¬ 
toes on the signals with low numbers of transits and low 
SNR here. This leaves six well-behaved light curves with 
a reasonable number of high SNR transits for which we 
have no ready explanation for their non-recovery. 

One comparison we can investigate is between the tar¬ 
gets on the nominal/best focus CCD channels, and the 
channels at the edge of the field, which experience the 
worst focus. Although the focus changes across the 
field of view, it is relatively stable across the scientific 
timescales of interest, cycling on a yearly basis with 
the spacecraft thermal environment. Indeed, we find 
that there is only a slight improvement in detection effi¬ 
ciency for the ‘best’ focus CCDs compared to those with 
the ‘worst’ focus. Examining the FGK sample as de¬ 
fined above, we find best fit coefficients of a = 4.26, 
b = 1.04 (a 27.6% recovery rate at 7.1er) and a = 4.79, 
b = 0.98 (a recovery rate of 22.7% at 7.1cr), respectively. 
This is distinct from the fact that the targets on the 
‘wor st’ focus channels typica lly have higher CDPP val¬ 
ues dChristiansen et all 1 2012 : and therefore it is already 
more difficult to detect low SNR planetary signals in 
these channels; this statement is that, for a given CDPP, 
the detection efficiency of the pipeline is lower on the 
channels that are less well focussed. 

We also examined the different electronic artefacts that 
were sampled (e.g. variable black/bias correction, FGS 
crosstalk, rolling band artefacts, and start-of-line ring¬ 
ing; see the footnotes of Table |T| for a description of these 
artefacts) and the results are delivered in Table [3] We 
find small variations in the recovery rate from the average 
rate for all FGK targets, with variable black/bias correc¬ 
tion and rolling band artefact channels having a slightly 
higher recovery rate, and the FGS crosstalk and start- 
of-line ringing channels having rates comparable to the 
non-electronic-artefact channels examined. At this stage 
we are hesitant to over-interpret the slightly higher rates 
in some channels as meaningful, due to the relatively 
small number of injections and recoveries per behaviour 
studied, and also since the periodicity of the noise intro¬ 
duced by these CCDs is, like the focus changes, approx¬ 
imately yearly, as the targets cycle off the CCD channel 
with the electronic artefact and typically onto more well- 
behaved CCDs, before returning to the noisier channel 
a year later. We can expect to uncover longer period 
changes in sensitivity when a longer observation baseline 
is examined in the next experiment. 

3.2. Recovery of injected parameters 

For the recovered injections, we can compare the values 
fitted by the pipeline to those injected. First, we examine 

1 This issue will be resolved in the 9.3 version of TPS. 
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the preservation of the injected transit depth through 
the pipeline, and find that the pipeline algorithms do 
a very good job of preserving the expected depth. We 
showed in Paper I that the first stage of the pipeline 
(generation of the aperture photometry and cotrending 
of the photometry to remove systematics) reduced the 
depths of the injected transits by a very small amount: 
the average measured SNR of the injected transits was 
99.7% of the injected SNR, which for individual transits 
(the focus of that paper) corresponds directly to a 99.7% 
recovery of the injected depths. Here, we take the ratio 
of the final fitted transit depth of the folded transit as 
reported by the Data Validation (DV) pipeline module 
to the average measured transit depth of the injected 
transits. In Figured we show this ratio as a function of 
the injected orbital period. There are two populations 
of points in this plot: those with periods longer than 10 
days, and those with shorter periods. 

A histogram of the ratio for the shorter period popu¬ 
lation is shown in panel (a) of Figure [5] also shown is 
a best-fit normal distribution, with a median value of 
97.0% and a standard deviation of 4.0%. However, the 
mean is only 87.3%; the main reason for the reduction 
in measured transit depth compared to injected transit 
depth is the harmonic filter, as discussed earlier in this 
section. There, we were specifically highlighting injec¬ 
tions which were not successfully recovered because the 
reduction in transit depth brought them below the detec¬ 
tion threshold of the pipeline; here we see the population 
of injections that are recovered, but at a reduced depth. 
This has stronger implications for the inferred size of the 
planet population at periods shorter than 10 days which 
should be folded into population analyses. 

The best-fit normal distribution to the longer period 
population is shown in panel (b) of Figure[5j with a mean 
value of 97.0% and a standard deviation of 5.0%. This 
indicates that the pipeline algorithms are reducing the 
depths of the longer period signals by ~3% in the latter 
stages, which include the harmonic filtering to remove 
sinusoidal signals, the whitening of the light curve to 
remove correlated noise, the normalising of the resulting 
harmonic-removed, whitened light curves, and the fitting 
of the transit model to the final full light curve product. 
This reduction results in a slight (~1.7%) decrease in 
the planet radius that would be measured from the light 
curve than what was injected; this is considerably smaller 
than the typical planet radius errors, but is a systematic 
average decrease and should be taken into account in 
robust population analyses. 

Although we have validated the pipeline algorithms 
here in preserving the transit depth, we caution that the 
results should not be interpreted as comprising the total 
error budget on the planet radius that would be inferred 
from the measured transit depth. There are at least two 
additional source of uncertainty to consider—the stel¬ 
lar parameters and the dilution of the transit depth by 
third light in the photometric aperture. The pipeline 
estimates a correction for this dilution by modelling the 
known sources in the aperture and subtracting the appro¬ 
priate flux, but this estimation is limited and constitutes 
an additional error on the planet radius calculation. 

We also compare the measured orbital period to the in¬ 
jected orbital period, and find an extremely tight distri¬ 
bution, shown in panel (c) of Figure [5] Fitting a normal 
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Figure 4. Distribution of transit depth recovery as a function of 
orbital period. The majority of points at periods longer than 10 
days lie around a value of 1, indicating that the pipeline is pre¬ 
serving the depths of the transits well. The population of points 
with periods shorter than 10 days shows a large scatter towards 
recovering shallower depths than are injected. This can be at¬ 
tributed largely to the action of the harmonic filter, which removes 
sinusoidal signals in the light curve before the data are searched 
for transit signals, and acts more strongly on short-period transit 
signals. 

distribution to the data gives a mean value of 1 and a 
standard deviation of 1.3e-5; it is therefore very unlikely 
that our criteria for a ‘successful’ recovery of matching 
the period to within 3% has eliminated real detections. 
At the longest periods examined here (~185 days), this 
corresponds to an uncertainty of 3 minutes. 

4. DISCUSSION 

The most significant impact of the detection efficiency 
is on the calculation of the underlying planet occurrence 
rates. As described in Section [I] previous analyses have 
typically assumed some thresh old or c u rve fo r the de¬ 
tection efficiency. iCatanzarite fc Shaol (|201 lh assumed 
100% completeness for signals with an SNR> 7.1cr for 
planets larger than 2R m and periods less than 130 days. 
iBorucki et all (I2011bl) assumed the perfect detector de¬ 
scribed in the previous section (50% completeness at 
SNR of 7.1er), for a signal with a minimu m of two tran¬ 
sits, o ut to orbital periods of 138 days. iHoward et all 
( 2012 1 assumed a 100% detection efficiency for transit¬ 
ing signals with an SNR > 10cr, for planets larger than 
2R®, and periods less than 50 days, based on the de¬ 
creasing rate of detection of these objects with time, 
and their readily apparent signals to the eye. lYoudinl 
(|2011l) makes the same assumption, and extends the pa¬ 
rameter space to plan ets down to OAR®. In a similar 
fashion, iDong fc Zhul (120131) assume a 100% detection 
efficiency for transit signals with an SNR > 8cr for plan¬ 
ets larger than 1R®, out to periods of 250 days (and 
found si milar results f or tra nsit signals with an SNR 
> 12cr). iFressin et all (120131) estimate the detection ef¬ 
ficiency from a comparison of the distribution of their 
modelled false positives with the planet candidates re¬ 
ported in iBatalha et all (|2013f ). finding a linear increase 
in detection efficiency from 0% at 6cr to 100% at 16cr, 
shown in Figure [3] as the magenta dashed line. Since 
these studies have examined different planet parameter 
spaces, it is difficult to isolate the impact of the choice 
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Figure 5. Comparisons of the injected and fitted parameters. 
Panel (a) shows the ratio of the fitted depth to the injected depth 
for injected transit signals with periods shorter than 10 days, panel 
(b) for injected transit signals with periods longer than 10 days, 
and panel (c) shows the ratio of the fitted period to the injected 
period across the entire period range. 


of detection efficiency on the derived occurrence rates. 

In order to examine this impact, we perform a toy- 
model occurrence rate analysis three times, varying only 
the assumption of detection efficiency in each case: for 
an optimistic detection efficiency assumption (the perfect 
detector, used by Borucki et al. 2011b), called Detection 
Efficiency 1 (DEI) for the remainder of the paper; the 
detection efficiency curve empirically measured in this 
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Figure 6. The distribution in period and radius of the 352 planet 
candidates selected for this study. The colour of the point corre¬ 
sponds to the reported SNR at which the planet candidate was 
detected. 


study, called DE2; and a pessimistic assumption (the 
linear ramp in probability o f detection fr om 0% at 6cr 
to 100% at 16cj described bv lFressin et al.1 (12013D . called 
DE3. All three of these detection efficiency curves are 
shown in Figure [5] 

Using the (closed) Q1-Q12 Kepler Object of Interest 
(KOI) table at the NASA Exoplanet Archive (Rowe et al. 
2015), we select all the objects classified as ‘Planet Can¬ 
didates’ with planet radii from 1-2 Rq, and orbital peri¬ 
ods from 10-320 dayt0, around the FGK stellar sample 
as defined above. This results in 352 planet candidates, 
shown in Figure[S] There are a large number of false pos¬ 
itives at periods < 10 days and > 320 days (see Fig. 8 of 
Tenenbaum et al. 2013), so we exclude those candidates 
in this sample. We caution strongly that the Q1-Q12 
KOI sample was not a uniformly-generated planet sam¬ 
ple (see Rowe et al. 2015 for more details), and we are 
using it solely as a starting point to illustrate the sys¬ 
tematic errors that arise from different assumptions in 
the detection efficiency, not as a starting point for deriv¬ 
ing robust occurrence rates. 

For each of the detection efficiency assumptions, we 
perform the following analysis, similarly to PHM13: 


1. For each planet in the sample, we calculate the ge¬ 
ometric correction, f g = a/R *, where a is the semi¬ 
major axis of the orbit, and i?* is the radius of the 
star the candidate is suspected to be transiting. 

2. Using the Q1-Q16 stellar table (the most up- 
to-date stellar parameters available, Huber et al. 
2014) from the NASA Exoplanet Archive, we then 
check, for each of the N s = 152,066 FGK stars with 
some observations between Q1 and Q12, whether 
this planet would have been detected around that 
star. We do this by comparing the transit depth 
of the planet around that star {R^/R 2 S ), where 
R p is the planet radius and R s is the stellar ra¬ 
dius from the aforementioned table, to the Q1-Q12 
root-mean-square (rms) of the CDPP value of that 

2 http://exoplanetarchive.ipac.caltech.edu, as of December 11th, 
2014 
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star, for the transit duration closest to that cal¬ 
culated for this planetQ. We then determine the 
average number of transits of the planet we could 
expect on that star, by using the orbital period and 
the distribution of observations obtained for that 
star over the twelve quarters, and calculate the to¬ 
tal expected SNR on average for this planet/star 
combination. 

3. Applying the relevant detection efficiency curve 
(DEI, DE2 or DE3) we determine the probability, 
p s , that this planet/star combination would have 
been detected by the pipeline, given that the planet 
transits the star. 

4. Thus for each planet, we find the fraction of stars 
around which it would have been detected, fd = 

Ps)/N s , by summing the probability for each 
star and dividing by the total number of stars. 

5. Finally, we correct each planet for its incomplete¬ 
ness (geometric and detectability), where C p = 
fg/fd is the corrected number of planets. 

We then calculate the occurr ence rates in a similar 
fashion to iHoward et al.l (120121 1 and PHM13, by per¬ 
forming the calculation at regularly-spaced grid points in 
planet radius and ^og(orbital period). We divide orbital 
period into bins of 0.5-1.25, 1.25-2.5, 2.5-5, 5-10, 10- 
20, 20-40, 40-80, 80-160 and 160-320 days, and planet 
radius into four equal bins from 1-2R0. In each bin, we 
calculate the total number of planets falling in that bin, 
using the corrected values described above, and divide 
by the total number of stars observed, N s . We use the 
Poisson uncertainties in the original number of planets 
falling into each bin to determine the final uncertainties. 

The differences caused by varying the detection effi¬ 
ciency assumptions manifest themselves in the derived 
occurrence rates. Figure 0 shows the results of the oc¬ 
currence rate calculation for each of the three detection 
efficiency curves tested. In each case, the 701 planet 
candidates are plotted in red. The grid in which the oc¬ 
currence rates are calculated is overlaid, and the colour 
of the box scales with the occurrence rate as measured 
in that box. The colour scale is fixed across all three im¬ 
ages so they can be compared directly. We caution again 
that these are not an attempt at robustly derived occur¬ 
rence rates; we have neglected many effects, such as the 
false positive rate of the planet sample, the uniformity of 
the planet sample, the reliability in the human vetting 
of the planet sample, and any multiplicity or systematic 
error in the parameters in the stellar sample. This is a 
simple analysis designed to examine the impact of the 
choice of detection efficiency curve on the derived occur¬ 
rence rates, and to attempt to quantify the underlying 
systematic errors that result. 

Qualitatively, the optimistic detection efficiency curve 
(DEI) assumes the pipeline detected all the planets it 
would have been expected to, and the derived occur¬ 
rence rate is based on the number found. In contrast, the 

3 Each star has a set of Combined Differential Photometric Preci¬ 
sion (CDPP) values, which are a measure of the noise in the target 
light curve as calculated by the Kepler pipeline in the whitened 
domain, for a set of time scales correspon din g to t ransit durations 
of interest, 1.5-15 hours UChristiansen et al.ll20l3 ). 


pessimistic efficiency curve (DE3) assumes the pipeline 
missed detections, and subsequently the number found 
is corrected to a higher ‘real’ value before the occurrence 
rate is calculated, resulting in a significantly higher de¬ 
rived occurrence rate of planets. The discrepancies are 
most significant for orbital periods in the range 5-80 
days, where the derived occurrence rates are > 3 a dis¬ 
crepant between the optimistic and pessimistic assump¬ 
tions. The empirical detection efficiency curve measured 
in this study lies between the optimistic and pessimistic 
curves, as shown in Figure [3] and this is reflected in 
the derived occurrence rates, which are systematically 
higher than the occurrence rates derived under the op¬ 
timistic case and lower than those under the pessimistic 
case across the entire period range. 

The differences are more significant when we integrate 
over the radius range under consideration and derive the 
total occurrence rates for planets with radii between 1— 
2R ffi . These integrated occurrence rates are given in 
Table 0] For periods shorter than 5 days, most tran¬ 
sit signals will have a very high expected MES, where 
the three detection efficiency curves converge, and in¬ 
deed we see very little difference in the derived occur¬ 
rence rates between the models. At longer periods, the 
occurrence rates derived with the measured detection 
efficiency curve are similar to (although systematically 
higher than) those with the optimistic curve, only diverg¬ 
ing at the longest periods due to the drop-off in average 
MES described above. However the occurrence rates de¬ 
rived with the pessimistic curve disagree at more than 3cr 
from the optimistic or measured curves for periods 5-160 
days; i.e. using the pessimistic detection efficiency curve, 
you would rule out the occurrence rate as derived with 
the measured detection efficiency curve presented here. 
This highlights the need for empirical measurement of 
the pipeline detection efficiency, and the continued re- 
evaluation of the pipeline performance as it is improved 
and applied to longer datasets; the pessimistic curve was 
derived from an earlier data set and earlier version of 
the pipeline, and it is not surprising that it is no longer 
applicable to the current planet population analyses. 

5. CONCLUSIONS AND FURTHER WORK 

We present here the first empirical measurement of the 
detection efficiency of the Kepler pipeline when detect¬ 
ing periodic transit signals, based on the injection and 
recovery of simulated transit signals injected into the cal¬ 
ibrated pixel data. With a baseline of one year, we find 
that the measured detection efficiency for FGK dwarfs 
(4000K < T e ff < 7000K, logg > 4.0) is well described by a 
T function with the coefficients a = 4.35, b = 1.05. How¬ 
ever, we know from examination of the Kepler pipeline 
detections for longer baselines (see, for example, Tenen- 
baum et al. 2013) that unexpected behaviour occurs at 
longer periods than those examined in this study, espe¬ 
cially periods 300-400 days, due to the annual rotation 
of the targets around the Kepler field of view. Since 
one of the primary goals for the Kepler Mission is the 
measurement of the occurrence rate of planets in the 
habitable zone of stars like the Sun, and these longer 
periods encompass that parameter space, it is impera¬ 
tive that we extend the analysis described here to longer 
baselines. We plan to run a comparable transit injection 
experiment for the full Kepler observing baseline (Ql- 
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Figure 7. The occurrence rates as calculated assuming the detection efficiency of the theoretical perfect detector (DEI, top), the measured 
efficiency (DE2, middle), and the linear ramp (DE3, bottom). The colours of the boxes scale with the occurrence rate in that box. 
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Q17) and derive the equivalent detection efficiency curve. 
One particular area of study will be the detectability of 
multi-planet systems- -the extent to which the presence 
of multiple periodic signals in the data and the order 
in which they are detected and removed by the pipeline 
before subsequent searches impacts the final detection 
efficiency 

Another process that needs to be quantified is the ex¬ 
amination of the pipeline candidates and subsequent clas¬ 
sification as either planet candidates or false positives. 
For the early catalogues (up to the Q1-Q12 catalogue, 
Rowe et al. 2015), this classification was done entirely by 
a team of humans, evaluating each candidate one by one. 
The project is moving towards more automated methods 
of classification by creating algorithms to automate the 
decision making process (McCauliff et al. 2015, Jenkins 
et al. in prep). The first steps toward a completely auto¬ 
mated process were taken in the Q16 catalogue (Mullally 
et al. 2015). The final vetting process, whether human- 
or machine-based, introduces an additional ‘detection ef¬ 
ficiency’, whereby some real planet candidates may not 
be promoted to planet candidate status and be incor¬ 
rectly classified as false positives. Our plan is to use the 
longer baseline run described above to also quantify the 
detection efficiency of the post-pipeline analysis, by re¬ 
producing the decision-making process as closely as pos¬ 
sible, and examining the rate at which our ‘real’ planets 


are discarded. 

Finally, we also have the capacity to inject the simu¬ 
lated transit signal at a location offset from the target 
star, allowing us to simulate false positive signals (i.e. 
due to eclipsing binaries along the line of sight). By al¬ 
locating some number of targets in the longer baseline 
experiment to study this, we will be able to examine the 
ability of the pipeline centroid analysis to identify and 
discard these false positives, and to identify the parame¬ 
ter space in which this identification is reliable. 
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Table 2 

Parameters of the injected transiting planets. The full table (10,341 rows) is available as an electronic data supplement to the paper. 


Kepler ID 

Sky group 

Period 

(days) 

Rp/Rs 

a/R s 

b 

^"depth 

(ppm) 

-^duration 

(hours) 

Epoch 

(BMJD) 

Expected MES 

Recovered 

9755118 

9 

79.9530 

0.0352 

91.2298 

0.5093 

1241.473 

5.206 

55708.382 

14.518 

i 

9755154 

9 

0.8870 

0.0778 

4.8476 

0.2943 

6047.453 

1.087 

55680.339 

260.376 

i 

9755234 

9 

57.9840 

0.0576 

77.2773 

0.2320 

3322.075 

4.457 

55668.172 

34.776 

i 

9815278 

9 

8.0340 

0.0453 

17.5103 

0.1890 

2055.826 

2.725 

55733.756 

133.925 

i 

9815334 

9 

1.7550 

0.0243 

2.5050 

0.5289 

591.887 

4.162 

55723.432 

164.277 

i 

9815427 

9 

1.8240 

0.0246 

4.8823 

0.8932 

606.471 

2.219 

55686.696 

115.277 

i 

9815482 

9 

35.9650 

0.0370 

55.5882 

0.9583 

1370.606 

3.843 

55654.269 

13.464 

i 

9815492 

9 

31.5930 

0.0489 

46.6409 

0.2202 

2390.012 

4.024 

55734.182 

62.903 

i 

9815530 

9 

109.7020 

0.0667 

132.7673 

0.2451 

4454.328 

4.908 

55649.244 

34.684 

i 

9815687 

9 

8.2130 

0.0146 

15.9514 

0.1131 

213.809 

3.058 

55731.764 

19.592 

i 

9815837 

9 

50.5360 

0.0360 

62.9225 

0.6047 

1296.469 

4.771 

55649.738 

27.238 

i 

9874912 

9 

18.9600 

0.0272 

34.5842 

0.2569 

738.354 

3.256 

55649.172 

20.623 

i 

9875034 

9 

65.3840 

0.0172 

38.8429 

0.8067 

295.809 

9.999 

55642.542 

11.171 

i 

9875070 

9 

28.5350 

0.0452 

52.5194 

0.0158 

2042.317 

3.227 

55685.497 

70.409 

i 

9875085 

9 

103.4400 

0.0140 

81.0096 

0.6069 

196.574 

7.585 

55700.170 

5.220 

0 

9875336 

9 

4.2350 

0.0180 

10.1619 

0.4792 

323.330 

2.476 

55704.299 

20.937 

1 

9875410 

9 

44.4410 

0.0163 

30.1883 

0.5857 

265.188 

8.745 

55647.623 

14.071 

1 

9875451 

9 

44.5600 

0.0552 

59.6182 

0.1615 

3044.727 

4.440 

55719.308 

40.660 

1 

9875707 

9 

1.2590 

0.0608 

5.9426 

0.6506 

3693.535 

1.258 

55724.124 

238.038 

1 

9875793 

9 

118.8520 

0.0270 

119.2722 

0.3738 

728.677 

5.919 

55651.631 

19.200 

1 
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Table 3 

r function best-fit coefficients across the focal plane. 


Targets 

Injections 

Recoveries 

a 

b 

R(7.1cr) 

All channels 

All targets 

10341 

9123 

4.21 

1.13 

23.6% 

FGK targets 

8579 

7696 

4.35 

1.05 

25.8% 

Non-FGK targets 

1762 

1427 

4.77 

1.24 

12.0% 

Subset ol channels, FGK targets 

Nominal/best focus 

1077 

953 

4.26 

1.04 

27.6% 

Edge of field/worst focus 

1732 

1553 

4.79 

0.98 

22.7% 

Variable black/bias correction 

1350 

1220 

3.88 

1.15 

29.1% 

Rolling band artefacts/Moire pattern drift 1961 

1750 

3.86 

1.12 

28.9% 

FGS crosstalk 

1977 

1791 

4.97 

0.91 

23.9% 

Start-of-line ringing 

482 

429 

7.72 

0.58 

18.0% 










Table 4 

Integrated occurrence rates (percentage of FGK stars with planets in this period range) across 1—2R.0, for the period ranges 0.5-1.25 days, 1.25-2.5 days, 2.5-5 days, 5-10 days, 

10-20 days, 20-40 days, 40-80 days, 80-160 days and 160-320 days. 


Detection Efficiency 

0.5-1.25d 

1.25—2.5d 

2.5-5d 

5-10d 

10—20d 

20-40d 

40-80d 

80-160d 

160-320d 

DEI (optimistic) 

DE2 (measured) 

DE3 (pessimistic) 

0.053±0.014% 

0.055±0.014% 

0.060±0.015% 

0.24±0.04% 

0.26±0.04% 

0.31±0.05% 

1.20± 0.11% 

1.33±0.12% 

1.59±0.15% 

2.90±0.22% 

3.36±0.25% 

4.33±0.32% 

5.26±0.40% 

6.38±0.48% 

8.81±0.66% 

5.15±0.54% 

6.43±0.67% 

9.25±0.97% 

6.27±0.85% 

8.17±1.10% 

12.72±1.71% 

9.62±2.21% 

13.11±3.01% 

22.42±5.15% 

5.37±1.69% 

7.29±2.31% 

12.34±3.90% 







